Previous | 
Home
Section: New Results
Capturing Deformations of Interacting Soft Objects Using RGB-D Data
We present a method for tracking multiple interacting deformable objects undergoing rigid motions, elastic deformations and contacts, using images and point cloud data provided by an RGB-D sensor (see Fig. 10). A joint registration framework is proposed, based on physical Finite Element Method (FEM) elastic and interaction models. It first relies on a visual segmentation of the considered objects in the RGB images. The different segmented point clouds are then processed to estimate rigid transformations with on an ICP algorithm, and to determine geometrical point-to-point correspondences with the meshes. External forces resulting from these correspondences and between the current and the rigidly transformed mesh can then be derived. It provides both non-rigid and rigid data cues. Classical collision detection and response model is also integrated, giving contact forces between the objects. The deformations of the objects are estimated by solving a dynamic system balancing these external and contact forces with the internal or regularization forces computed through the FEM elastic model. This approach has been here tested on different scenarios involving two or three interacting deformable objects of various shapes, with promising results.
A case study in open surgery on the liver has also been investigated. Yet in this case a major improvement in the accuracy of the registration is provided by the integration of anatomical shape constraints, which are naturally hidden from the RGB-D camera, and that we account for through a registration with the pre-operative CT data. With a comparative study, we demonstrate the relevance of our method in a real-world application mimicking an open surgery scenario where the liver has to be tracked to provide an augmented reality view.
The method has been implemented through software developed within the team, especially the RGBDTracking plugin. 6.4 and 6.3. Validation has been performed on porcine in-vivo data, acquired in accordance with UE norms, in collaboration with Pr. Mario GIMENEZ and Dr. Alain GARCIA, Pr. Federico Davrieux, and Pim Hendriks, Daan Kuppens and from IHU-Strasbourg.
